{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using multi-armed bandits to choose the best model for predicting credit card default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [helm](https://github.com/helm/helm)\n",
    "- [minikube](https://github.com/kubernetes/minikube) --> install 0.25.2\n",
    "- [s2i](https://github.com/openshift/source-to-image)\n",
    "\n",
    "- Kaggle account to download data.\n",
    "- Python packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Either head to https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset or use the Kaggle API (instructions at https://github.com/Kaggle/kaggle-api) to download the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kaggle datasets download -d uciml/default-of-credit-card-clients-dataset\n",
    "!unzip default-of-credit-card-clients-dataset.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and inspect data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv('UCI_Credit_Card.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = 'default.payment.next.month'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[target].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we have a class imbalance, so if we use accuracy as the performance measure of a classifier, we need to be able to beat the \"dummy\" model that classifies every instance as 0 (no default):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[target].value_counts().max()/data.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Case study for using multi-armed bandits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In deploying a new ML model, it is rarely the case that the existing (if any) model is decommissioned immediately in favour of the new one. More commonly the new model is deployed alongside the existing one(s) and the incoming traffic is shared between the models.\n",
    "\n",
    "Typically A/B testing is performed in which traffic is routed between existing models randomly, this is called the experiment stage. After a set period of time performance statistics are calculated and the best-performing model is chosen to serve 100% of the requests while the other model(s) are decommissioned.\n",
    "\n",
    "An alternative method is to route traffic dynamically to the best performing model using multi-armed bandits. This avoids the opportunity cost of consistently routing a lot of traffic to the worst performing model(s) during an experiment as in A/B testing.\n",
    "\n",
    "This notebook is a case study in deploying two models in parallel and routing traffic between them dynamically using multi-armed bandits (Epsilon-greedy and Thompson sampling in particular)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the dataset to simulate a real-world scenario consisting of several steps:\n",
    "\n",
    "1. Split the data set in half (15K samples in each set) and treat the first half as the only data observes so far\n",
    "2. Split the first half of the data in proportion 10K:5K samples to use as train:test sets for a first simple model (Random Forest)\n",
    "3. After training the first model, simulate a \"live\" environment on the first 5K of data in the second half of the dataset\n",
    "4. Use the so far observed 20K samples to train a second model (XGBoost)\n",
    "5. Deploy the second model alongside the first together with a multi-armed bandit and simulate a \"live\" environment on the last 10K of the unobserved data, routing requests between the two models\n",
    "\n",
    "The following diagram illustrates the proposed simulation design:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![data-split](assets/split.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "OBSERVED_DATA = 15000\n",
    "TRAIN_1 = 10000\n",
    "TEST_1 = 5000\n",
    "\n",
    "REST_DATA = 15000\n",
    "\n",
    "RUN_DATA = 5000\n",
    "ROUTE_DATA = 10000\n",
    "\n",
    "# get features and target\n",
    "X = data.loc[:, data.columns!=target].values\n",
    "y = data[target].values\n",
    "\n",
    "# observed/unobserved split\n",
    "X_obs, X_rest, y_obs, y_rest = train_test_split(X, y, random_state=1, test_size=REST_DATA)\n",
    "\n",
    "# observed split into train1/test1\n",
    "X_train1, X_test1, y_train1, y_test1 = train_test_split(X_obs, y_obs, random_state=1, test_size=TEST_1)\n",
    "\n",
    "# unobserved split into run/route\n",
    "X_run, X_route, y_run, y_route = train_test_split(X_rest, y_rest, random_state=1, test_size=ROUTE_DATA)\n",
    "\n",
    "# observed+run split into train2/test2\n",
    "X_rest = np.vstack((X_run, X_route))\n",
    "y_rest = np.hstack((y_run, y_route))\n",
    "\n",
    "X_train2 = np.vstack((X_train1, X_test1))\n",
    "X_test2 = X_run\n",
    "\n",
    "y_train2 = np.hstack((y_train1, y_test1))\n",
    "y_test2 = y_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train both models at once, but defer evaluation of the second model until simulating the live environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rf = RandomForestClassifier(random_state=1)\n",
    "rf.fit(X_train1, y_train1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how good our first model is on the test1 set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, \\\n",
    "f1_score, confusion_matrix, classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_preds1 = rf.predict(X_test1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(classification_report(y_test1, y_preds1,\n",
    "                           target_names=['No default','Default']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for score in [accuracy_score, precision_score, recall_score, f1_score, confusion_matrix]:\n",
    "    print(score.__name__ + ':\\n', score(y_test1, y_preds1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from utils import plot_confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(y_test1, y_preds1)\n",
    "plot_confusion_matrix(cm, classes=['No default','Default'], normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So a simple random forest model without any optimizations is able to outperform random guessing on accuracy and achieves a baseline F1 score of ~0.44. However, it is a poor predictor of default as it only achieves a recall of ~0.34."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the second model in advance, but defer evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "xgb = XGBClassifier(random_state=1)\n",
    "xgb.fit(X_train2, y_train2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save trained models to disk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.externals import joblib\n",
    "joblib.dump(rf, 'models/rf_model/RFModel.sav')\n",
    "joblib.dump(xgb, 'models/xgb_model/XGBModel.sav')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up Kubernetes for live simulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick Kubernetes cluster on GCP or Minikube."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minikube = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if minikube:\n",
    "    !minikube start --vm-driver kvm2 --memory 4096 --cpus 6\n",
    "else:\n",
    "    !gcloud container clusters get-credentials standard-cluster-1 --zone europe-west1-b --project seldon-demos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a cluster-wide cluster-admin role assigned to a service account named “default” in the namespace “kube-system”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin \\\n",
    "--serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add current context details to the configuration file in the seldon namespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create tiller service account and give it a cluster-wide cluster-admin role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check deployment rollout status and deploy seldon/spartakus helm charts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-operator --name seldon-core --set usageMetrics.enabled=true   --namespace seldon-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/seldon-controller-manager -n seldon-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install stable/ambassador --name ambassador --set crds.keep=false"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deployment.apps/ambassador"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install analytics (Prometheus for metrics and Grafana for visualisation):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-analytics --name seldon-core-analytics \\\n",
    "    --set grafana_prom_admin_password=password \\\n",
    "    --set persistence.enabled=false \\\n",
    "    --namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Port forward Ambassador (run command in terminal):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```sh\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app.kubernetes.io/name=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Port forward Grafana (run command in terminal):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```sh\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') -n seldon 3000:3000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then view an analytics dashboard inside the cluster at http://localhost:3000/d/rs_zGKYiz/mab?refresh=1s&orgId=1&from=now-2m&to=now. Login with:\n",
    "\n",
    "Username : admin\n",
    "\n",
    "password : password (as set when starting seldon-core-analytics above)\n",
    "\n",
    "**Import the mab dashboard from ```assets/mab.json```.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wrap model and router images with s2i"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have prepared the model classes under ```models/rf_model/RFModel.py``` and ```models/xgb_model/XGBModel.py``` for wrapping the trained models as docker images using s2i. The structure of the files is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pygmentize models/rf_model/RFModel.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we define our own custom metrics which are the entries of the confusion matrix that will be exposed to Prometheus and visualized in Grafana as the model runs in the simulated live environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If Minikube used: create docker image for the trained models and routers inside Minikube using s2i."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if minikube:\n",
    "    !eval $(minikube docker-env) && \\\n",
    "    make -C models/rf_model build && \\\n",
    "    make -C models/xgb_model build && \\\n",
    "    make -C ../epsilon-greedy build && \\\n",
    "    make -C ../thompson-sampling build"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the first model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f assets/rf_deployment.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/rf-model-rf-model-7c4643e "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulate the first model in production for 5000 samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import rest_request_ambassador, send_feedback_rest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(X_run.shape[0]):\n",
    "    if i%1000 == 0:\n",
    "        print(f'Processed {i}/{X_run.shape[0]} samples', flush=True)\n",
    "        \n",
    "    # fetch sample and make a request payload\n",
    "    x = X_run[i].reshape(1,-1).tolist()\n",
    "    request = {'data':{'ndarray':x}}\n",
    "\n",
    "    # send request to model\n",
    "    response = rest_request_ambassador('rf-deployment', 'seldon', request)\n",
    "\n",
    "    # extract prediction\n",
    "    probs = response.get('data').get('ndarray')[0]\n",
    "    pred = np.argmax(probs)\n",
    "\n",
    "    # send feedback to the model informing it if it made the right decision\n",
    "    truth_val = int(y_run[i])\n",
    "    reward = int(pred==truth_val)\n",
    "    truth = [truth_val]\n",
    "    _ = send_feedback_rest('rf-deployment', 'seldon', request, response, reward, truth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the model performance on the Grafana dashboard:\n",
    "http://localhost:3000/d/rs_zGKYiz/mab?refresh=1s&orgId=1&from=now-2m&to=now (refresh to update)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy the original model and the new model with a router in front"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose now we have come up with a new model and want to deploy it alongside the first model with a multi-armed bandit router to make decisions which model should make predictions. We will delete the original deployment and make a new one that has both models in parallel and a router/multi-armed bandit in front.\n",
    "\n",
    "To make things interesting, we will actually deploy 2 parallel deployments with the same 2 models but a different router in front (Epsilon-greedy and Thompson sampling) to compare the performance of two very different multi-armed bandit algorithms. One can think of the first deployment as a production deployment and the second parallel one as a shadow deployment whose responses are used for testing only."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But first, let's see what the performance of the new XGBoost model is on its test2 data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_preds2 = xgb.predict(X_test2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(classification_report(y_test2, y_preds2,\n",
    "                           target_names=['No default','Default']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for score in [accuracy_score, precision_score, recall_score, f1_score, confusion_matrix]:\n",
    "    print(score.__name__ + ':\\n', score(y_test2, y_preds2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cm = confusion_matrix(y_test2, y_preds2)\n",
    "plot_confusion_matrix(cm, classes=['No default','Default'], normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the XGBoost model is slightly better than the old RFModel, so we expect any decent multi-armed bandit router to pick this up on live data, let's try this out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, delete the existing deployment of the old RFModel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl delete sdep rf-deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deploy the following two deployments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import get_graph\n",
    "get_graph('assets/eg_deployment.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_graph('assets/ts_deployment.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl apply -f assets/eg_deployment.json -n seldon\n",
    "!kubectl apply -f assets/ts_deployment.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/poc-eg-eg-2-47fb8da"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl rollout status deploy/poc-ts-ts-2-75f9d39"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simulate both deployments in parellel with the remaining 10000 data samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we send request and feedback to both parallel deployments, thus assessing the performance of the Epsilon-greedy router versus Thompson sampling as a method of routing to the best performing model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(X_route.shape[0]):\n",
    "    if i%1000 == 0:\n",
    "        print(f'Processed {i}/{X_route.shape[0]} samples', flush=True)\n",
    "        \n",
    "    # fetch sample and make a request payload\n",
    "    x = X_route[i].reshape(1,-1).tolist()\n",
    "    request = {'data':{'ndarray':x}}\n",
    "\n",
    "    # send request to both deployments\n",
    "    eg_response = rest_request_ambassador('eg-deployment', 'seldon', request)\n",
    "    ts_response = rest_request_ambassador('ts-deployment', 'seldon', request)\n",
    "\n",
    "    # extract predictions\n",
    "    eg_probs = eg_response.get('data').get('ndarray')[0]\n",
    "    ts_probs = ts_response.get('data').get('ndarray')[0]\n",
    "    eg_pred = np.argmax(eg_probs)\n",
    "    ts_pred = np.argmax(ts_probs)\n",
    "\n",
    "    # send feedback to the model informing it if it made the right decision\n",
    "    truth_val = int(y_route[i])\n",
    "    eg_reward = int(eg_pred==truth_val)\n",
    "    ts_reward = int(ts_pred==truth_val)\n",
    "    truth = [truth_val]\n",
    "    _ = send_feedback_rest('eg-deployment', 'seldon', request, eg_response, eg_reward, truth)\n",
    "    _ = send_feedback_rest('ts-deployment', 'seldon', request, ts_response, ts_reward, truth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the model performance on the Grafana dashboard:\n",
    "http://localhost:3000/dashboard/db/mab?refresh=5s&orgId=1 (refresh to update)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We note that both the Epsilon greedy and Thompson sampling allocate more traffic to the better performing model (XGBoost) over time, but Thompson Sampling does so at a quicker rate as evidenced by the superior metrics (F1 score in particular)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete data\n",
    "!rm default-of-credit-card-clients-dataset.zip\n",
    "!rm UCI_Credit_Card.csv\n",
    "\n",
    "# delete trained models\n",
    "!rm models/rf_model/RFModel.sav\n",
    "!rm models/xgb_model/XGBModel.sav\n",
    "\n",
    "# delete Seldon deployment from the cluster\n",
    "!kubectl delete sdep --all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
